Factorization Machines Factorized Polynomial Regression Models

Factorization Machines (FM) are a new model class that combines the advantages of polynomial regression models with factorization models. Like polynomial regression models, FMs are a general model class working with any real valued feature vector as input for the prediction of real-valued, ordinal or categorical dependent variables as output. However, in contrast to polynomial regression models, FMs replace two-way and all other higher order interaction effects by their factorized analogues. The factorization of higher order interactions enables efficient parameter estimation even for sparse datasets where just a few or even no observations for those higher order effects are available. Polynomial regression models without this factorization fail. This work discusses the relationship of FMs to polynomial regression models and the conceptual difference between factorizing and non-factorizing model parameters of polynomial regression models. Additionally, we show that the model equation of factorized polynomial regression models can be calculated in linear time and thus FMs can be learned efficiently. Apart from polynomial regression models, we also focus on the other origin of FMs: factorization models. We show that the standard factorization models matrix factorization and parallel factor analysis are a special case of FMs and additionally how recently proposed and successfully applied factorization models like SVD++ can be represented by FMs. The drawback of typical factorization models is that they are not applicable for general prediction tasks but work only for categorical input data. Furthermore their model equations and optimization algorithms are derived individually for each task. By knowing that FMs can mimic these models just by choosing the appropriate input data, we finally conclude that deriving a learning algorithm, e.g., Stochastic Gradient Descent (SGD) or Alternating Least Squares (ALS), for FMs once is sufficient to get the learning algorithms for all factorization models automatically, thus saving a lot of time and effort.